Lithopone is rather nontoxic, due to the insolubility of its components. It has been used in medicine as a radiocontrast agent. Lithopone is allowed to be in contact with foodstuffs in the US and Europe.[1]
As for titanium dioxide, the FDA approved titanium dioxide for use as a food additive in 1966. The last time the agency reviewed the additive’s safety, according to the Guardian, was in 1973.
Although barium sulfate is almost completely inert, zinc sulfide degrades upon exposure to UV light, leading to darkening of the pigment. The severity of this UV reaction is dependent on a combination of two factors; how much zinc sulfide makes up the pigments formulation, and its total accumulated UV exposure. Depending on these factors the pigment itself can vary in shade over time, ranging from pure white all the way to grey or even black. To suppress this effect, a dopant may be used, such as a small amount of cobalt salts, which would be added to the formulation. This process creates cobalt-doped zinc sulfide. The cobalt salts help to stabilize zinc sulfide so it will not have as severe a reaction to UV exposure.
Certificate of Analysis (Lithopone B301, Lithopone B311 powder TDS)
While the FDA maintains that the regulated use of titanium dioxide is safe, the European Food Safety Authority and some other experts warn of potential, serious health risks.